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(54) Unsteadiness compensation in solid state Imagers 

(57) An apparatus for reducing the effect of smear in a TDl image detector comprises a charge accumulation 
device (304) which stores a row (308) of electrical charges representing a sensed image, the charges being 
blurred due to motion of the electronic imager. The imager includes a shift register (302) for generating a 
blurred video line by concatenating the blurred charges and an image unsteadiness compensator (312) which 
receives motion information from sensors (314) representing motion of the electronic imager. The image 
unsteadiness compensator (312) computes image wander of the blurred charges by using the motion 
information and generates a corrected video line from the blurred video iine by using the image wander 
computation. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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Electronic Image Unsteadiness Compensation 



The present invention relates generally to camera imagers (such as 
photographic cameras), and, more particularly, to an electronic system and 
method for compensating for image unsteadiness in camera imagers. 



In using camera imagers to electronically collect sensed images, the 
length of time during which a patch of object space is imaged onto a 
corresponding angularly-sized pixel is referred to as the exposure time. The 
signal level generated by the camera imager is directly proportional to this 
exposure time. Thus, all other things being equal, longer exposure times 
produce larger signal levels. Larger signal levels have better pixel-to-pixel 
signal-to-noise ratios relative to those having shorter exposure times. 

However, as exposure time is increased, two problems arise. First, the 
area of the scene which can be covered or scanned per unit time becomes 
restricted, sometimes to the point of diminished system utility. Second, the 
amount of image smear resulting from vehicle perturbations may increase. If 
this image smear is not compensated for, it may reduce image quality. 

Conventional techniques have solved the first problem by using time 
delay and integration (TDI) detectors. TDI detectors effectively increase 
available exposure time by accumulating a larger amount of charge while 



eliminating the penalty of minimizing coverage. This 
desirable result is obtained because a sensor using a 
number (N) of TDI stages can scan N times as fast as a 
corresponding linear detector. These multiple stages 
enable multiple coverage of the same portion of the scene, 
resulting in a charge which is proportional to the number 
of TDI stages used. Thus, a TDI detector provides 
approximately N times the angular coverage of a 
corresponding linear detector which utilizes the same total 
exposure time. The use of TDI detectors, however, does not 
solve the second problem. Specifically, the use of TDI 
detectors does not eliminate the effects of image smear 
(also referred to as image unsteadiness) as a function of 
total exposure time. 

Conventional systems which have been employed to 
stabilize unsteadiness and avoid image smear use mechanical 
or optomechanical methods to stabilize the sensor with 
respect to the object space. In many instances, however, 
it is difficult to achieve the desired tolerance levels. 
Subsequently, the exposure time and number of TDI 
integrators which may be effectively employed are 
negatively impacted. This can often be the limiting factor 
on system performance and mission satisfaction. Therefore, 
an effective and efficient system and method for 
compensating for image unsteadiness is required. 

In accordance with one aspect of the present 
invention, a method for correcting a blurred video line 
generated by an electronic imager having a charge 
accumulation device for storing rows of electrical charges 
representing a sensed image, the electrical charges blurred 
due to motion of the electronic imager, wherein the blurred 
video line comprises pixel values proportional to the 
blurred charges in one of the rows comprises the steps of: 

(1) receiving motion information from sensors, said 
motion information describing the motion of the electronic 
imager ; 
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(2) computing image wander of the blurred charges by 
using said motion information; and 

(3) generating a corrected video line by using said 
image wander computation to correct said blurred video 

5 line. 

In accordance with a second aspect of the present 
invention, an electronic imager comprises: 

(a) charge accumulation means for storing a row of 
electrical charges representing a sensed image, said 

10 charges blurred due to motion of the electronic imager; 

(b) means, coupled to said charge accumulation means, 
for generating a blurred video line by concatenating said 
blurred charges, wherein said blurred video line comprises 
pixel values proportional to said blurred charges; 

15 (c) means, coupled to said charge accumulation means, 

for correcting said blurred video line, said correcting 
means comprising: 

(1) means for receiving motion information from 
sensors, wherein said motion information describes said 

2 0 motion of the electronic imager; 

(2) means for computing image wander of said 
blurred charges by using said motion information; and 

(3) means for generating a corrected video line 
from said blurred video line by using said image wander 

2 5 computation . 

In one embodiment, an electronic imager has a charge 
accumulation device, such as a TDI detector. The charge 
accumulation device stores a row of electrical charges 
representing a sensed image. The charges are blurred due 

30 to motion of the electronic ^ a cf or 
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relative to the image scene which leads to image wander and contamination of 
pixels. The resulting TDI detector output is a blurred video line. 

The imager includes a shift register for generating a blurred video line 
by concatenating the blurred charges. The blurred video line includes a series 
5 of pixel values proportional to the blurred charges. 

The imager also includes an image unsteadiness compensator for 
correcting the blurred video line. The image unsteadiness compensator 
generally operates as follows. First, the image unsteadiness compensator 
receives motion information from sensors, wherein the motion information 
10 describes the motion of the image falling on the electronic imager (due to the 

motion of the electronic imager). Then, the image unsteadiness compensator 
computes image wander of the blurred charges by using the motion 
information. Next, the image unsteadiness compensator generates a corrected 
video line from the blurred video line by using the image wander computation. 



15 According to a preferred embodiment, the image unsteadiness 

compensator, as generally described above, is implemented according to the 
following theory. The blurred line is considered to be a convolution of the 
corrected line arid a blur kernel. The blurred line is known. Therefore, the 
image unsteadiness compensator generally operates by first determining the 

20 blur kernel. Then, the image unsteadiness compensator uses the blur kernel 

to perform a deconvolution of the blurred line in order to generate the 
corrected line. 
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The specification describes two ennbodiinents for performing the 
deconvolution. The first embodiment involves a spatial approach. The second 
embodiment involves a frequency approach. 



Further features and advantages of the present invention, as well as the 
structure and operation of various embodiments of the present invention, are 
described in detail below with reference to the accompanying drawings. In the 
drawings, like reference numbers indicate identical or functionally similar 
elements. Additionally, the left-most digit of a reference number identifies the 
drawing in which the reference number first appears. 



The present invention will be described with reference to the 
accompanying drawings, wherein: 

Figure 1 illustrates a conventional camera imager; 

Figure 2 illustrates the manner in which a modulation transfer function 
(MTF) at the Nyquist frequency is degraded by image smear; 

Figure 3 illustrates a camera imager according to the present invention; 

Figure 4 illustrates the general operation of the image unsteadiness 
compensator of the present invention; 

Figure 5 illustrates portions of an example corrected line, blurred line, 
and blur kernel; 

Figure 6 illustrates in greater detail the operation of the image 
unsteadiness compensator of the present invention; 

Figure 7 graphically illustrates a portion of a blurred video line 316; 

Figure 8 illustrates a column of a blurred line; 

Figure 9 illustrates a flow chart of the operation of a spatial approach 
of the present invention; 

Figure 10 illustrates an example matrix equation which is used to 
calculate a set of linear equations for use with the spatial approach; and 

Figures 11 and 12 collectively illustrate a flow chart of a frequency 
approach of the present invention. 



Figure 1 illustrates the effects of image smear in a camera imager 100. 
Camera imager 100 includes a shift register 102, a TDI detector 104, and an 
amplifier 106. TDI detector 104 includes multiple stages 108A through 108E 
(collectively and generally referred to as 108). The stages include multiple 
columns 110A through 110J (collectively and generally referred to as 110). 
An image produced by a scanning sensor (not shown in Figure 1) enters the 
TDI detector 104 at the bottom stage 108E and proceeds smoothly toward the 
top stage 108A. The charges are moved in nominal synchronism with the 
image so that five times the total charge is generated (as compared to a single 
linear detector) by the time the image has traversed all five of the TDI stages 
108 shown. 

The summed charge is then transferred to shift register 102 and 
subsequently output to amplifier 106 during the interval required for the next 
line of image charges to reach the shift register 102 from TDI detector 104. 
The entire process is pipeline in nature such that each pixel always has some 
part of an image incident upon it. The mechanism by which the charges are 
synchronized with the image usually involves a multi-phase transfer process 
which approximates a completely smooth motion of the charge in the direction 
indicated by arrow 112. 

The ideal motion of an unperturbed image patch is shown in column 
HOC by image path 120. The image following image path 120 moves 
smoothly in both direction (and time) from the point of entering the TDI 
detector 104 to the point where it is transferred into shift register 102. This 
represents what would occur in the absence of all image unsteadiness. 



7 



What occurs in the presence of uncompensated image unsteadiness is 
referred to as image wander and is represented in Figure 1 by image path 102. 
The associated image patch is not confined to a single column (that is, column 
1 10H), but wanders and contaminates columns 1 10G and 1 101 as well. Since 
5 all portions of an image scene which are adjacent to each other are essentially 

moving together, columns HOG and 1101 contaminate column 110H in a 
similar fashion. 

Image smear reduces the modulation transfer function (MTF) of a 
camera imager. MTF is the Fourier transform of the line spread function. 
10 Image scheme reduces MTF in accordance with the expression given below 

in equation (1). 

= Sin 7C 9 tfR (Equation 1) 

*etfR 

where: © = image unsteadiness (radians/second) 

t = total (integrated) exposure time (seconds) 
f = focal length (millimeters) 
15 R = spatial frequency (cycles/mm) 

Image smear is introduced by a number of disturbances. For example, 
if the platform on which the camera imager is located is in an aircraft, these 
disturbances include roll, pitch, yaw, forward motion compensation (FMQ, 
angle of attack, and drift angle. These disturbances are well known in the art. 

20 Figure 2 illustrates the manner in which a modulation transfer function 

at the Nyquist frequency is degraded by image smear. The general form of 
the curve is that of a sine function. At an image smear of one pixel, the 
Nyquist MTF is reduced to 63%. With a smear of two pixels, the Nyquist 
MTF is zero. Thus, the system limitation to correcting image wander is the 
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signal-to-noise ratio at each pixel. The amount of image wander may not 
exceed ±2 pixels if useful compensation is to be achieved. According to the 
preferred embodiment, the present invention reduces the effective image smear 
to one half a pixel or less. Thus, as shown in Figure 2, the Nyquist MTF is 
90% at a total smear of one half pixel. 

Figure 3 illustrates a camera imager 300 according to the present 
invention. Camera imager 300 includes a shift register 302, a TDI detector 
304, an amplifier 306, and an image unsteadiness compensator 312. TDI 
detector 304 contains multiple stages 308A through 308D (collectively and 
generally referred to as 308). Stages 308 are divided into columns 310A 
through 310J (collectively and generally referred to as 310). In the preferred 
embodiment of the present invention, TDI detector 104 has four stages 108A- 
108D. This has been found to be the optimal number of stages to reduce the 
impact on MTF. However, as one skilled in the relevant art would know, 
TDI detector 104 may have any number of stages. 

The image unsteadiness compensator 312 receives motion information 
on a motion data line 320 from sensors 314. The structure and operation of 
shift register 302, TDI detector 304, and amplifier 306 are well known in the 
art and operate as described with reference to Figure 1 . The general operation 
of image unsteadiness compensator 312 shall now be described with reference 
to the flowchart illustrated in Figure 4. 

Step 402 indicates the initiation of the image smear correction process. 
In step 404, image unsteadiness compensator 312 receives a blurred video line 
over line 316 from amplifier 306. Blurred video line 316 includes a series of 
pixel values which are proportional to the charges in shift register 302, 
Blurring of the video line 316 is due to the platform disturbances described 
above. 



In step 406, the image unsteadiness compensator 312 computes the 
degree of image wander by using the motion information transmitted on 
motion data line 320 from sensors 314. The image wander was described 
above with reference to columns HOG, 110H and 1101 in Figure 1. For 
example, image wander may occur when camera imager 301 is located on an 
airborne platform, the motion of which causes the image wander. The sensors 
304, which may be well-known gyros, sense the motion of the platform. The 
motion information transmitted on motion data line 320 quantifies this platform 
motion. 

In step 408, the image unsteadiness compensator 312 generates a 
corrected video line 318 by using a image wander computation (discussed 
below) to correct blurred video line 316. The image smear correction process 
is then completed at step 410. 

According to the preferred embodiment, image unsteadiness 
compensator 312, as generally described above, is implemented according to 
the following theory. 

The blurred video line 316 is considered to be a convolution of a 
corrected video line 318 and a blur kernel (not shown in Figure 3). This is 
illustrated in Figure 5, which illustrates portions of an example corrected video 
line 318, blurred video line 316, and a blur kernel 502. Each element of the 
blurred video line 316 is determined by convolving two elements of corrected 
video line 318 with two elements of the blur kernel 504 (for example, 5*1 + 
4*1 = 9 and 4*1 + 2*1 = 6). 

The blurred video line 316 is transmitted from amplifier 306. 
Therefore, image unsteadiness compensator 312 generally operates by first 
determining blur kernel 502. Then, image unsteadiness compensator 312 uses 
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this blur kernel to perform a deconvolution of the blurred video line 316 in 
order to generate the corrected video line 318. 

At each TDI stage, the theoretical center of the pixel is offset from the 
center of the focal place element as a function of the orientation of the aircraft 
and of the servo errors. The output line consists of the sum of these shifted 
TDI stages. It is assumed that the error can be decomposed into a constant x 
shift (parallel to the long axis of the focal plane) and a y shift which varies 
with distance from the center of the focal plane. 

Maximum value for 8^ was assumed to be 1,800 ^radians/second. 
For 16 TDI stages and a 1 millisecond exposure, the total FMC error will be 
28.8 /iradians. The roll servo error, 5 roU = 500 /iradians/second. For 16 TDI 
stages and a 1 millisecond exposure, the total roll error will be 8.0 ^radians. 
The rotational error, 5 rot , is assumed to be 900 /^radians/second, or 14.4 
^radians during the exposure. The analysis does not include terms for 
uncompensated angle of attack or drift, under the assumption that they are 
removed by the pointing mirror. 

The image motion errors may be reduced to "X" motion components, 
"Y" motion components, and rotational components (having both "X M and " Y n 
terms). According to the present invention, the image motion errors are 
represented according to two equations (2) and (3) below. 

5 X = f Sfmc (Equation 2) 



'cot 



(Equation 3) 



where: 



'FMC 



error in Forward Motion Component 
error in Roll 



'roll 



5, 



'rot 



error in Rotation 



f 

X 



These calculations must be repeated for each TDI stage used, for each 
line and the results accumulated. 

The maximum 8 X shift, using these values, is 1.2 pixels and the 
maximum S y shift is 0.3 pixels at the center of the array and 0.4 pixels at the 
ends. 

Since the 5 y is a function of the Z coordinate, the shift changes across 
the focal plane, theoretically necessitating a different convolution kernel for 
each pixel. Instead, we can pick a tolerance of 0.1 pixel smear, and since the 
maximum Y shift is less than 1 pixel we can divide each half of the focal 
plane into 10 regions, each of which has the same shift within the region. The 
shift will be symmetrical on each side of the array center. 

As Equations (2) and (3) indicate, S x is represented by a single X 
motion component: forward motion correction (FMC). Rotational components 
in the X direction are negligible. Thus, S x does not vary among the pixels in 
the blurred video line 316. 

5 y is represented by a Y motion component (that is, roll) and a 
rotational component. The roll component does not vary among pixels in the 
blurred video line 316. However, the rotational component does vary with 
pixels (that is, position) among the line 308. 

Equations (2) and (3) do not include terms for all disturbances, such 
as angle of attack and drift (which are assumed to be removed by a pointing 
mirror). These disturbances result in some geometric distortion but do not 



significantly degrade resolution. However, Equations (2) and (3) could be 
easily extended to include terms for other disturbances by using well-known 
mathematical relationships for these disturbances. 

Figure 6 is a flowchart further illustrating step 406, the computation 
of image wander, performed by image unsteadiness compensator 312. Step 
406 is illustrated as the entry point into this sequence of steps in Figure 6. 
At this point, the image unsteadiness compensator 312 has received blurred 
video line 316 from the amplifier 306 as illustrated in step 404 of Figure 4. 

In step 602, the image unsteadiness compensator 312 uses the motion 
information received from sensors 314 over motion data line 320 to calculate 
the motion and rotational components (that is, 6 mc , 5 roU and 5 rot ). 

In step 604, the image unsteadiness compensator 312 uses the results 
of step 602 to compute 6 X for blurred video line 316 (described below). 

According to the preferred embodiment, the blurred video line 502 is 
divided into a number of regions. In step 606, the image unsteadiness 
compensator 312 uses the results of step 602 to compute a 5 y value is 
computed for each region. Specifically, for each region, a pixel from the 
region is selected to calculate the 8 y value for the region (preferably, the center 
pixel in each region is used). In the preferred embodiment of the present 
invention, there are 10 regions. 

In step 608, image unsteadiness compensator 312 computes an X blur 
kernel (described below). The X blur kernel is used in subsequent steps to 
correct the blurred video line 316 for disturbances in the X direction. The 
manner in which the X blur kernel is calculated will be described with 



reference to Figure 7, which graphically illustrates a portion of the blurred 
video line 316. 

Referring to Figures 5 and 7, blurred video line 316 has pixels PI, P2, 
P3, P4, P5, P6 and P7 with values 5, 4, 2, 1, 6, 7 and 8, respectively. 
Assume that AX (as calculated in step 604) is 1.5 pixels. This means that the 
pixels have been shifted by 1.5 pixels in the X direction. Therefore, P3 has 
a blurred value of 2. Its unblurred value is 3.5, which represents a shift of 
1.5 pixels. Likewise, P4 has a blurred value of 1. Moving 1.5 pixels to the 
right, it is evident that P4 has an unblurred value of 6.5. The shift of 1.5 
pixels is equivalent to a deconvolution with a blur kernel of 0,0,0,0,0,.5,.5. 

Returning to Figure 6, in step 610, image unsteadiness compensator 
312 calculates a Y blur kernel for each region of blurred video line 316. The 
Y blur kernel in a region is calculated in a manner similar to which the X blur 
kernel is calculated (in step 608), except y values are used to calculate the Y 
blur kernel. The Y blur kernels are used in subsequent steps to correct the 
blurred video line 316 for disturbances in the Y direction. 

The manner in which the Y blur kernel for a region of the line 316 is 
calculated will now be described in greater detail with reference to Figure 8. 
Figure 8 illustrates a portion of the blurred line 316. This blurred line has a 
region which contains 5 pixel values, that is, UC1, UC2, UC3, UC4 and 
UC5. In calculating the Y blur kernel for this region, the image unsteadiness 
compensator 312 uses the middle pixel, which in this case, is UC3. To 
calculate the Y blur kernel for this region, the image unsteadiness compensator 
312 uses values from previous video lines 316P and future lines 316F which 
are in the same column 310 as pixel UC3. 
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As shown in Figure 8, UC A , UC B , UC C , and UC D from blurred column 
310 are used to calculate the Y blur kernel for the region containing UC3 in 
the blurred video line 316. UC A and UC B exist in lines 316P which occurred 
before blurred line 316. UC c and UC D exist in lines 316F which occurred 
5 after the blurred line 316. Therefore, in order to calculate the blur kernel for 

the region in the blurred line 316, the image unsteadiness compensator 312 
must maintain a memory of lines 316P which occurred before the blurred line 
316 and lines 316F which occurred after the blurred line 316. According to 
the preferred embodiment of the present invention, image unsteadiness 
10 compensator 312 includes a memory sufficient to store 15 lines 316. Thus, 

the image unsteadiness compensator 312 calculates the blurred kernel for each 
region using 7 previous lines 316P and 7 future lines 316F. As evident from 
the above discussion, the image unsteadiness compensator 312 cannot 
compensate the blurred line 316 until after the future lines 316F occur. 

15 In step 408, correction of blurred video lines image unsteadiness 

compensator 312 uses the X and Y blur kernels calculated in the previous 
steps to correct blurred video line 316 to generate corrected video line 318. 
The present invention includes two embodiments for performing step 408: a 
spatial approach and a frequency approach. These are described in detail in 

20 the following sections with reference to Figures 9-12. Step 408, correct 

blurred video lines, is illustrated as the entry point into each of these 
flowcharts. 

A. Spatial Approach 

Figure 9 illustrates a flow chart of the operation of the spatial approach 
25 of the present invention. The spatial approach views the deconvolution 

problem as a set of linear equations and performs a matrix solution to obtain 
corrected video line 318. 



In step 902, image unsteadiness compensator 312 uses the pixel values 
of blurred video line 316 and the X blur kernel (calculated in step 608 of 
Figure 6) to generate a set of linear equations for the corrected video line 318 
in the X direction. Figure 10 illustrates an example matrix equation which is 
used to calculate the set of linear equations. The matrix equation shown in 
Figure 10 includes a coefficient matrix 1002, a variable array 1004, and a 
results array 1006. 

The coefficients matrix 1002 is a J by K matrix, wherein J is equal to 
the number of pixels in each TDI stage 308. In the preferred embodiment, J 
is 4000. K is equal to the size of the variable array 1004. Each row of the 
coefficients matrix 802 includes the X blur kernel (K^ and K 2 , for example) 
followed and/or preceded by a series of zeros. 

The variable array 1004 has a size equal to J + 1. The variables C u 
C 2 , . . C 10 in variable array 1004 represent the pixel values of corrected 
video line 318. As shown in Figure 10, variable array 1004 includes a 
dummy value D. Dummy value D is required to implement the matrix 
equation shown in Figure 10 (since the X blur kernel is two digits (that is, K, 
and Kj), the coefficients matrix 1002 J dimension is not equal to its K 
dimension). 

The results array 1006 has a size equal to J. The values of the results 
array (that is, UQ, UC 2 , . . ., UCjq) represent the pixel values from the 
blurred video line 316. 

In step 902, therefore, image unsteadiness compensator 312 performs 
well-known matrix algebra operations on the matrices shown in Figure 10 in 
order to generate the set of linear equations. 



In step 904, image unsteadiness compensator 312 solves the set of 
linear equations for the pixel values of the corrected video line 318 (that is, 
the variables in the variable array 1004) with regard to disturbances in the X 
direction. The following steps in the spatial approach of the present invention 
operate to correct the blurred video line 316 with regard to disturbances in the 

Y direction. 

In step 906, image unsteadiness compensator 312 selects one of the 
regions of the blurred line 316. In step 908, image unsteadiness compensator 
312 selects one of the pixels in the selected region. In steps 910 and 912, 
image unsteadiness compensator 312 operates to correct the pixel selected in 
step 908 for disturbances in the Y direction. 

Specifically, in step 910, image unsteadiness compensator 312 
generates a set of linear equations for the selected pixel in the Y direction. 
The set of linear equations are generated using a matrix equation as shown in 
Figure 10. In this instance, however, the coefficient matrix 1002 includes the 

Y blur kernel for the region selected in step 906. Also, the variables (that is, 
Q, Q2, . . C 10 ) in the variable array 1004 represent the corrected values of 
the pixels in the column 310 containing the pixel selected in step 908 (see 
Figure 8). The values (that is, UQ, UCj, . . UC 10 ) in the results array 
1006 represent the blurred values of the pixels in the column 310 associated 
with the pixel selected in step 908 (see Figure 8). 

As noted above, image unsteadiness compensator 312 compensates the 
blurred line 316 in the Y direction using previous lines 316P and future lines 
316F. In the preferred embodiment, the image unsteadiness compensator 312 
uses an equal number of previous lines 316P and future lines 316F. 
Therefore, the corrected value of the pixel selected in step 608 is represented 
by the middle variable in the variable array 1004. Also, the element of the 



blurred line 316 corresponding to the pixel selected in step 908 is represented 
by the middle element of the results array 1006. 

In step 912, the image unsteadiness compensator 312 solves the 
equations in order to determine the values of the variables in the variable array 
1004. All of the values are discarded except the value corresponding to the 
pixel selected in 908. This value is the value which has been corrected in the 
X and Y directions. The image unsteadiness compensator 312 outputs this 
value in its appropriate position in the corrected line 318. 

As indicated by step 914, steps 908, 910 and 912 are executed for each 
pixel in the region. As indicated by step 916, steps 908, 910, 912 and 914 are 
executed for each region in blurred video line 316. 

In the preferred embodiment of the present invention, the linear matrix 
operations of the spatial approach are performed by general purpose 
computers. Such general purpose computers are well known to those skilled 
in the art. 

B. Frequency Approach 

Figures 11 and 12 collectively illustrate a flow chart of the frequency 
approach used in the present invention to correct blurred video line 316. The 
frequency approach views the deconvolution problem as a problem in the 
frequency domain. Thus, according to the frequency approach, the 
deconvolution of the blurred video line 316 is performed using Wiener 
filtering in the frequency domain. The Wiener filtering requires Fourier 
transform processing. According to the preferred embodiment, the Fourier 
transform processing is implemented using Fourier processors. Such Fourier 
processors are well known to those skilled in the relevant art. 
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Referring now to Figures 11 and 12, in step 1102 the image 
unsteadiness compensator 312 calculates C(f), which is the digital Fourier 
transform (DFT) of blurred video line 316. 

In step 1104, image unsteadiness compensator 312 estimates N(f), 
which is the DFT of noise. According to the preferred embodiment, a white 
noise model is used. Therefore, no actual calculations are required to 
calculate N(f). Rather, N(f) is a constant amplitude. 

In step 1106, image unsteadiness compensator 312 uses C(f) and N(f) 
to calculate S(f), which is the DFT of blurred video line 316 minus the DFT 
of the noise. In step 1108, image unsteadiness compensator 312 calculates a 
filter function $(f) according to the following equation: 

I SCO 1 2 

*(f) — (Equation 4) 

|S(f)| 2 + |N(f)| 2 

In step 1110, image unsteadiness compensator 312 calculates R(f), 
which is the DFT of the X blur kernel of blurred video line 316. In step 
1112, image unsteadiness compensator 312 calculates U(f), which is the DFT 
of the corrected line 318. This is done according to the following equation: 

C(f)*(f) 

0(0 = (Equation 5) 

R(f) 

In step 11 14, image unsteadiness compensator 312 performs an inverse 
DFT on U(f) to obtain u(t), which represents the blurred video line 316 
corrected in the X direction (hereinafter called the partially corrected blurred 
video line 316). The remaining steps of the frequency approach are performed 



in order to correct the partially corrected blurred video line 316 in the Y 
direction. 

In step 1116, image unsteadiness compensator 312 selects a region of 
partially corrected blurred video line 316. In step 1118, image unsteadiness 
compensator 312 calculates R(f), which is the DFT of the Y blur kernel for 
the region selected in step 916. In step 1120, image unsteadiness compensator 
312 selects one of the pixels in the selected region. Then, image unsteadiness 
compensator 312 calculates C(f) in step 1122, which is the DFT of the blurred 
column which contains the selected pixel. For example, in Figure 8, if UQ 
is the selected pixel, then during step 1122, the image unsteadiness 
compensator 312 would calculate the DFT of blurred column 310. 

Next, processing continues through connector CI 124, image 
unsteadiness compensator 312 estimates N(f) in step 1202, which is the DFT 
of noise. This step is similar to step 1104 described above. In step 1204, 
image unsteadiness compensator 312 calculates S(f), which is the DFT of the 
blurred column (containing the selected pixel) minus noise. In step 1206, 
image unsteadiness compensator 312 calculates *(f) according to Equation (4) 
above. Next, image unsteadiness compensator 312 calculates U(f), which is 
the DFT of the unblurred column containing the selected pixel, in step 1208. 
This is done according to Equation (5), above. 

In step 1210, image unsteadiness compensator 312 performs an inverse 
DFT on U(f) to obtain u(t), which is the unblurred column. In step 1212, 
image unsteadiness compensator 312 selects the value from the unblurred 
column u(t) corresponding to the pixel selected in step 1120. This value, 
representing the corrected value of the pixel selected in step 1 120, is output 
as part of the corrected video line 318. 



CLAIMS 



1. A method for correcting a blurred video line generated 
by an electronic imager having a charge accumulation device 
for storing rows of electrical charges representing a 
sensed image, the electrical charges blurred due to motion 
of the electronic imager, wherein the blurred video line 
comprises pixel values proportional to the blurred charges 
in one of the rows, the method comprising the steps of: 

(1) receiving motion information from sensors, said 
motion information describing the motion of the electronic 
imager ; 

(2) computing image wander of the blurred charges by 
using said motion information; and 

(3) generating a corrected video line by using said 
image wander computation to correct said blurred video 
line. 

2. The method of claim 1, wherein step (2) comprises the 
steps of: 

(a) computing a forward motion correction error by 
using said motion information; 

(b) calculating an X position change of said blurred 
charges by using said forward motion correction error; and 

(c) computing an X blur kernel by using said X 
position change. 

3. The method of claim 2, wherein step (3) comprises the 
step of deconvolving said blurred video line to form said 
corrected video line by using said X blur kernel. 

4. The method of claim 3, wherein said deconvolving step 
comprises the steps of: 

(i) forming a coefficients matrix comprising elements 
equal to said X blur kernel; 



21 



(ii) forming a variable array comprising J variables, wherein said 
variables comprise pixel values of said corrected video line; 

(iii) forming a results array comprising J known values, wherein 
said known values comprise said pixel values of said blurred video line; 

(iv) generating a set of linear equations according to a matrix 
algebra equation: 

(coefficients matrix) * (variable array) = results array; and 

(v) solving said linear equations for said variables. 

5. The method of claim 3, wherein said deconvolving step comprises the 
steps of: 

(i) calculating C(f), representing a Fourier transform of said 
blurred video line; 

(ii) estimating N(f), representing a Fourier transform of noise; 

(iii) calculating S(f), representing a Fourier transform of said blurred 
video line minus said noise; 

(iv) calculating *(f), representing a filter function, according to a 
first equation: 

I SCO I 3 

*(f) = 

|S(f)| 2 + I N(0 I 2 

(v) calculating R(f), representing a Fourier transform of said X blur 

kernel; 

(vi) calculating U(f), representing a Fourier transform of said 
corrected video line, according to a second equation: 



C(f)*(f) 

0(f) = 

R(f) 
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(vii) calculating u(f), representing said corrected video line, by 
performing an inverse Fourier transform on U(f). 

6. A method according to any of the preceding claims, further 

comprising the step of storing video information representing the 
sensed image. 

7. The method of claim 6, wherein said storing step comprises the steps 
of: 

storing video lines occurring before said blurred video line; 

storing said blurred video line; and 

storing video lines occurring after said blurred video line. 

8. The netted cf claim 6 or claim 7, vterein stag? (2) mr pi-im? the gi«=i« of: 
dividing the blurred video line into one or more regions, wherein each 

of said regions comprises one or more of the pixel values; and 
computing a Y blur kernel for each of said regions. 

9. The method of claim 8, wherein said step for computing a Y blur 
kernel for each of said regions comprises the steps of: 

(a) computing a roll error; 

(b) selecting one of said regions; 

(c) selecting one of the pixel values in said selected region; 

(d) computing a rotation error for said selected region by using an 
X position of said selected pixel value; 

(e) calculating a Y position change for said selected region by 
adding said roll error and rotation error; 

(0 computing a Y blur kernel for said selected region by using said 
Y position change; and 

(g) performing steps (b)-(f) for each of said regions. 
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10. The method of claim 8, wherein step (3) comprises the step of 
deconvolving said blurred video line to form said corrected video line by using 
said Y blur kernel. 

11. The method of claim 10, wherein said deconvolving step comprises the 
steps of: 

(a) selecting one of said regions; 

(b) selecting one of the pixel values in said selected region, wherein 
said selected pixel value has an X position; 

(c) forming a coefficients matrix comprising elements equal to said 
Y blur kernel associated with said selected region; 

(d) forming a variable array comprising J variables, wherein said 
variables comprise pixel values of said corrected video line; 

(e) forming a results array comprising J known values, wherein 
said known values comprise said selected pixel value and pixel values from 
said video information having said X position; 

(f) generating a set of linear equations according to a matrix 
algebra equation: 

(coefficients matrix) * (variable array) = results array; 

(g) solving said linear equations for said variables; 

(h) performing steps (b)-(g) for each of said pixels in said selected 
region; and 

(i) performing steps (a)-(h) for each of said regions. 

12. The method of claim 10, wherein said deconvolving step comprises the 
steps of: 

(a) selecting one of said regions; 

(b) selecting one of the pixel values in said selected region, wherein 
said selected pixel value has an X position; 
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(c) calculating C(f), representing a Fourier transform of a blurred 
column comprising said selected pixel value and pixel values from said video 
information having said X position; 

(d) estimating N(f)> representing a Fourier transform of noise; 

(e) calculating S (f) , representing a Fourier transform of said blurred 
column minus said noise; 

(f) calculating #(f), representing a filter function, according to a 
first equation: 

I S(f) | 2 

*(f) = 

|S(f)| 2 + |N(f)| 2 

(g) calculating R(f), representing a Fourier transform of said Y blur 
kernel associated with said selected region; 

(h) calculating U(f), representing a Fourier transform of a corrected 
column, according to a second equation: 

C(f)*(0 
0(f) = 

R(f) 

(i) calculating u(f), representing said corrected column, by performing 
an inverse Fourier transform on 0(f); 

(j) performing steps (b)-(i) for each of said pixels in said selected 
region; and 

(k) performing steps (a)-(j) for each of said regions. 

13, An electronic imager, comprising: 

(a) charge accumulation means for storing a row of electrical 
charges representing a sensed image, said charges blurred due to motion of the 
electronic imager; 
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(b) means, coupled to said charge accumulation means, for 
generating a blurred video line by concatenating said blurred charges, wherein 
said blurred video line comprises pixel values proportional to said blurred 
charges; 

5 ( c ) means, coupled to said charge accumulation means, for 

correcting said blurred video line, said correcting means comprising: 

(1) means for receiving motion information from sensors, 
wherein said motion information describes said motion of the electronic 
imager; 

10 (2) means for computing image wander of said blurred 

charges by using said motion information; and 

(3) means for generating a corrected video line from said 
blurred video line by using said image wander computation. 

1 5 14. The electronic imager of claim 13, wherein said means for computing 
image wander comprises: 

means for computing a forward motion correction error by using said 
motion information; 

means for calculating an X position change of said blurred charges by 
20 using said forward motion correction error; and 

means for computing an X blur kernel by using said X position change. 

15. The electronic imager of claim 14, wherein said means for generating 
a corrected video line comprises means for deconvolving said blurred video 
line to form said corrected video line by using said X blur kernel. 

25 

16. The electronic imager of claim 15, wherein said deconvolving means 
comprises: 

(i) means for forming a coefficients matrix comprising elements 
equal to said X blur kernel; 
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(ii) means for forming a variable array comprising J variables, 
wherein said variables comprise pixel values of said corrected video line; 

(iii) means for forming a results array comprising J known values, 
wherein said known values comprise said pixel values of said blurred video 

5 line; 

(iv) means for generating a set of linear equations according to a 
matrix algebra equation: 

(coefficients matrix) * (variable array) = results array; and 

(v) means for solving said linear equations for said variables. 

10 17. The electronic imager of claim 15, wherein said deconvolving means 
comprises: 

(i) means for calculating C(f), representing a Fourier transform of 
said blurred video line; 

(ii) means for estimating N(f)» representing a Fourier transform of 

15 noise; 

(iii) means for calculating S(f), representing a Fourier transform of 
said blurred video line minus said noise; 

(iv) means for calculating $(f), representing a filter function, 
according to a first equation: 

20 I S(f) | 2 

*(0 = 

|S(f)| 2 + |N(f)|* 

(v) means for calculating R(f), representing a Fourier transform of . 
said X blur kernel; 

(vi) means for calculating U(f), representing a Fourier transform of 
25 said corrected video line, according to a second equation: 



C(f)*(f) 
0(f) 
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R(f) 

(vii) means for calculating G(f), representing said corrected video line, 
by performing an inverse Fourier transform on U(f). 

18. Pn ^iprrtr rpi^ imayar aoacaxLing to ay of claim s 13 to 17, further 

5 comprising means for storing video information representing the sensed 
image* 

19. The electronic imager of claim 18, wherein said storing means 
comprises: 

means for storing video lines occurring before said blurred video line; 
means for storing said blurred video line; and 
10 means for storing video lines occurring after said blurred video line. 

20. Hie electronic imager of claim 18 or claim 19, wherein said means 
for conputing image wander comprises: 

means for dividing the blurred video line into one or more regions, 
wherein each of said regions comprises one or more of the pixel values; and 
15 means for computing a Y blur kernel for each of said regions. 

21. The electronic imager of claim 20, wherein said means for computing 
a Y blur kernel for each of said regions comprises: 

means for computing a roll error; 

means for selecting one of said regions; 
20 means for selecting one of the pixel values in said selected region; 

means for computing a rotation error for said selected region by using 
an X position of said selected pixel value; 

means for calculating a Y position change for said selected region by 
adding said roll error and rotation error; and 
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means for computing a Y blur kernel for said selected region by using 
said Y position change. 

22. The electronic imager of claim 20, wherein said means for generating 
a corrected video line comprises means for deconvolving said blurred video 

5 line to form said corrected video line by using said Y blur kernel. 

23. The electronic imager of claim 22, wherein said deconvolving means 
comprises: 

means for selecting one of said regions; 

means for selecting one of the pixel values in said selected region, 
10 wherein said selected pixel value has an X position; 

means for forming a coefficients matrix comprising elements equal to 
said Y blur kernel associated with said selected region; 

means for forming a variable array comprising J variables, wherein 
said variables comprise pixel values of said corrected video line; 
15 means for forming a results array comprising J known values, wherein 

said known values comprise said selected pixel value and pixel values from 
said video information having said X position; 

means for generating a set of linear equations according to a matrix 
algebra equation: 

20 (coefficients matrix) * (variable array) = results array; and 

means for solving said linear equations for said variables. 

24. The electronic imager of claim 22, wherein said deconvolving means 
comprises: 

means for selecting one of said regions; 
25 means for selecting one of the pixel values in said selected region, 

wherein said selected pixel value has an X position; 



29 



means for calculating C(f), representing a Fourier transform of a 
blurred column comprising said selected pixel value and pixel values from said 
video information having said X position; 

means for estimating N(f), representing a Fourier transform of noise; 

means for calculating S(f), representing a Fourier transform of said 
blurred column minus said noise; 

means for calculating $(f), representing a filter function, according to 
a first equation: 

| SCO 1 2 

*(f) = — 

|S(f)| 2 + |N(f)T 

means for calculating R(f), representing a Fourier transform of said Y 
blur kernel associated with said selected region; 

means for calculating 0(f), representing a Fourier transform of a 
corrected column, according to a second equation: 

C(f)$(Q 

0(f) = ; and 

R(0 

means for calculating u(f), representing said corrected column, by 
performing an inverse Fourier transform on 0(f)- 

25. The electronic imager of claim 24, wherein said means for generating 
a corrected video line comprises one or more general purpose processors for 
performing matrix algebra. 

26. The electronic imager of claim 24, wherein said means for generating 
a corrected video line comprises one or more Fourier processors for 
performing Fourier transforms and inverse Fourier transforms. 
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27. A method for correcting a blurred video line 
substantially as hereinbefore described with reference to 
any of the examples shown in Figures 3 to 12 of the 
accompanying drawings. 

28, An electronic imager substantially as hereinbefore 
described with reference to any of the examples shown in 
Figures 3 to 12 of the accompanying drawings. 



Patents Act 1977 -> ^ I 
Examiner's report to the Comptroller under Section 17 J 
(The Search report) J 


Application number 
GB 9420595.2 


Relevant Technical Fields 

(i) UK CI (Ed.M) H4F - FHH, FGY, FESA 

(ii) Int CI (Ed.5) H04N - 3/15, 5/21, 217; G06F - 15/68 

Databases (see below) 

(i) UK Patent Office collections of GB, EP, WO and US patent 
specifications. 

(ii) ONLINE DATABASES - WPI, INSPEC 




Search Examiner'' '£ f 
D H JONES n ... 


*-*V i • : - ■ • 
Date of completion of Search 

15 DECEMBER 1994,- 




Documents considered relevant 




~Mlowing"a~searclrin respect of 
Claims :- 
1-28 



Categories of documents 

X: Document indicating lack of novelty or of inventive step. P: 

Y: Document indicating lack of invenUve step if combined with 

one or more other documents of the same category. E: 

A: Document indicating technological background and/or state 

of the art. &: 



Document published on or after the declared priority date 
but before the filing date of the present application. 

Patent document published on or after, but with priority date 
earlier than, the filing dale of the present application. 

Member of the same patent family; corresponding document. 



Category 


Identity of document and relevant passages 


Relevant to 
claim (s) 


X 


WO 92/10059 Al (RECON/OPTICAL) - see whole document 


1, 6, 7, 13, 
18, 19 



Databases:The UK Patent Office database comprises classified collections of GB. EP. WO and US patent specifications as outlined periodically in the Official Journal 
(Patents). The on-line databases considered for search are also listed periodically in the Official Journal (Patents). 



T7 - 15167 Page 1 of 1 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ iKlAGE CUT OFF AT TOP, BOTTOM OR SIDES 



□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




FADED TEXT OR DRAWING 



